Skip to content

Add PiecewiseSequence index array#8799

Merged
robert3005 merged 6 commits into
developfrom
codex/piecewise-sequential-array
Jul 17, 2026
Merged

Add PiecewiseSequence index array#8799
robert3005 merged 6 commits into
developfrom
codex/piecewise-sequential-array

Conversation

@danking

@danking danking commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

This array allows FixedSizeList, List, and ListView to implement take without materializing a dense array of indices of size O(N_TAKE_INDICES * LIST_SIZE). Their elements child arrays can, in turn, use efficient memcpy calls instead of copying individual indices.

Summary

Adds PiecewiseSequenceArray, a serializable index encoding for piecewise arithmetic sequences: starts[i] + j * multipliers[i] for j in 0..lengths[i].

This PR only introduces the index array and its execution/scalar behavior. It intentionally does not specialize take; dependent PRs consume this encoding for the constant-multiplier-1 contiguous case.

Validation

  • cargo fmt -p vortex-array
  • cargo test -p vortex-array piecewise_sequence

@github-actions

github-actions Bot commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 2b6fa3e 1 Explore Profiling Data
Previous Runs (7)
Status Commit Job Attempt Link
🟢 Done 52affcd 1 Explore Profiling Data
🟢 Done 16ca5dc 2 Explore Profiling Data
🟢 Done 6eb0c92 1 Explore Profiling Data
🟢 Done ba7ef2e 1 Explore Profiling Data
🟢 Done 1748e31 1 Explore Profiling Data
🟢 Done 3edf6c8 1 Explore Profiling Data
🟢 Done aba7e4d 1 Explore Profiling Data

Powered by Polar Signals Cloud

@codspeed-hq

codspeed-hq Bot commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will improve performance by 11.17%

⚡ 1 improved benchmark
✅ 1669 untouched benchmarks
⏩ 52 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Simulation true_count_vortex_buffer[128] 580.6 ns 522.2 ns +11.17%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

Comparing codex/piecewise-sequential-array (2b6fa3e) with develop (3975895)

Open in CodSpeed

Footnotes

  1. 52 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@github-actions

github-actions Bot commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +5.3%
Engines: DataFusion No clear signal (+6.1%, low confidence) · DuckDB No clear signal (+4.6%, low confidence)
Vortex (geomean): 1.047x ➖
Parquet (geomean): 1.020x ➖
Shifts: Parquet (control) +2.0% · Median polish +4.5%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
datafusion / vortex-file-compressed (1.080x ➖, 0↑ 0↓)
name PR 2b6fa3e (ns) base 3975895 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 10021365 9303558 1.08
vortex_q01/datafusion:vortex-file-compressed 6627376 6119813 1.08
datafusion / parquet (1.018x ➖, 0↑ 0↓)
name PR 2b6fa3e (ns) base 3975895 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 20583727 20667041 1.00
vortex_q01/datafusion:parquet 4789370 4604281 1.04
duckdb / vortex-file-compressed (1.069x ➖, 0↑ 0↓)
name PR 2b6fa3e (ns) base 3975895 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 10436535 9980727 1.05
vortex_q01/duckdb:vortex-file-compressed 6503703 5950906 1.09
duckdb / parquet (1.022x ➖, 0↑ 0↓)
name PR 2b6fa3e (ns) base 3975895 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23652549 23222387 1.02
vortex_q01/duckdb:parquet 9552799 9308681 1.03

No file size changes detected.

@danking
danking force-pushed the codex/piecewise-sequential-array branch 2 times, most recently from 3edf6c8 to 1748e31 Compare July 16, 2026 18:20
@danking danking mentioned this pull request Jul 16, 2026
Closed
@danking danking changed the title Add PiecewiseSequential index array Add PiecewiseSequence index array Jul 16, 2026
robert3005
robert3005 reviewed Jul 16, 2026
View reviewed changes
Comment thread vortex-array/src/arrays/piecewise_sequence/mod.rs
ctx: &mut ExecutionCtx,
) -> VortexResult<(PrimitiveArray, PrimitiveArray, PrimitiveArray)> {
let starts = array.starts().clone().execute::<PrimitiveArray>(ctx)?;
let lengths = array.lengths().clone().execute::<PrimitiveArray>(ctx)?;

@robert3005 robert3005 Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think while starts could be primitive we specifically want to handle cases where this is not primitive?

@danking danking Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, agreed. This function in particular is meant as the fallback for, e.g. if some user just constructs this array directly and uses it somewhere.

You're correct that the subsequent PRs lost the constant-length special cases. I'll add that back to #8800.

danking added 4 commits July 16, 2026 19:36
@danking
Add PiecewiseSequential index array
98e82f4 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
Rename PiecewiseSequential to PiecewiseSequence
40fe633 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
Add PiecewiseSequence multipliers
0e0fe56 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
Fix PiecewiseSequence rustdoc link
16ca5dc 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
danking force-pushed the codex/piecewise-sequential-array branch from 6eb0c92 to 16ca5dc Compare July 16, 2026 23:36
@danking danking added the changelog/feature A new feature label Jul 16, 2026
danking added 2 commits July 16, 2026 20:13
@danking
Use imported UnsignedPType in PiecewiseSequence
52affcd 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
Shorten PiecewiseSequence doc link
2b6fa3e 
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking
danking marked this pull request as ready for review July 17, 2026 15:25
@robert3005

robert3005 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

I think this is a good intermediary state. I have realised that #8825 would be the best solution but that is a lot bigger refactor than adding this

@robert3005
robert3005 merged commit c5e075a into develop Jul 17, 2026
83 checks passed
@robert3005
robert3005 deleted the codex/piecewise-sequential-array branch July 17, 2026 23:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robert3005 robert3005 robert3005 approved these changes
@joseph-isaacs joseph-isaacs Awaiting requested review from joseph-isaacs

Assignees

No one assigned

Labels

changelog/feature A new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@danking @robert3005